{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 壁画效果预览\n",
    "以下第一张为原图，第二张为经过Style1风格迁移后的效果图，第三张为经过Style2风格迁移后的效果图。总体来说，效果还是可以的。\n",
    "![原图](https://ai-studio-static-online.cdn.bcebos.com/7823b0bfca694cff8fd4681facbd5058d308048944ee434eae2160033889ec02)\n",
    "![Style1_tranfer](https://ai-studio-static-online.cdn.bcebos.com/a9a0eb6e343a4610a46cb6624f297710ded2e3cfe13d48a69f6aad4272232060)\n",
    "![Style2_tranfer](https://ai-studio-static-online.cdn.bcebos.com/f56e6e24c46941fea5bf96f9bc27028b93ef4704b69b45219f3581014346a288)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 背景介绍\n",
    "\n",
    "&emsp;&emsp;壁画，墙壁上的艺术，即人们直接画在墙面上的画。作为建筑物的附属部分，壁画的装饰和美化功能使它成为环境艺术的一个重要方面。**壁画为人类历史上最早的绘画形式之一**。  \n",
    "\n",
    "&emsp;&emsp;壁画是指绘在建筑物的墙壁或天花板上的图案。它分为粗底壁画、刷底壁画和装贴壁画等。壁画是最古老的绘画形式之一。至今埃及、印度、巴比伦、中国等文明古国保存了不少古代壁画。在意大利文艺复兴时期，壁画创作十分繁荣，产生了许多著名的作品。我国自周代以来，历代宫室乃至墓室都有饰以壁画；随着宗教信仰的兴盛，壁画又广泛应用于寺观、石窟（例如**敦煌莫高窟**、**芮城永乐宫**等）。我国至今仍大量保存着著名的佛教壁画和道教壁画遗迹。这些遗迹有部分已经被列入了**世界文化遗产的保护名录，作为我们古代文明的见证**。  \n",
    "\n",
    "&emsp;&emsp;我国陕西咸阳秦皇宫壁画残片，距今有2300年。唐代是我国壁画的兴盛时期，那段时期是我国壁画艺术的高峰期，创作出了很多古今闻名的壁画，如敦煌壁画、克孜尔石窟等。到了宋代以后，壁画逐渐衰落。直到1949年后，中国壁画才逐渐得到恢复与发展 。  \n",
    "\n",
    "&emsp;&emsp;随着现代社会生活多元化、个性化的发展，艺术上出现了更多的以“**表现自我**”为中心的艺术形式，现代壁画与建筑环境艺术为了顺应社会发展的需要，出现了多种多样的艺术形式，为人们创造了一个多元化的文化环境。正壁画艺术伴随着人类文明的发展，具有辉煌而悠久的历史，它是人类追求美的理想、表现内心精神世界的独特艺术形式。随着人们审美观念的变化，城市化进程的不断加快，现代建筑环境对壁画的要求，壁画语言发生了较大的变化，现代壁画无论从表现形式、语言内涵到其社会功能都与传统壁画有着较大差异。  \n",
    "\n",
    "&emsp;&emsp;现代壁画随着社会的发展也在不断的进步与发展，在互联网没有普及的岁月，人们更青睐于写实的、具象的、内容繁冗丰富的，但随着互联网的诞生与普及，人们受着过量信息的困扰，内心变得疲惫，需要的是放松、明快、简约的东西被再一次的肯定，**壁画领域也随之而改变，不断地推陈出新，来符合人们的需要与发展** 。  \n",
    "\n",
    "&emsp;&emsp;在建筑业蓬勃发展的今天，大多数建筑承袭现代主义“**少就是多**”的观念，外墙装饰均是清一色的玻璃幕墙或单色瓷砖，面目冷淡，缺乏感情，这种为追求工业的高速发展而生产的建筑形态，与人们向往的自然生态环境格格不入。然而，壁画可作为人们面对自然的一个窗口，能在大厦的外墙和中厅等看到描绘自然的壁画，无疑是一种心旷神怡的心灵感觉。同时，在各种壁画形式中，最耐久、最易清洗、最耐侵蚀、色彩最鲜艳，表现手法最多样的，当属陶瓷壁画。  \n",
    "\n",
    "&emsp;&emsp;故而在现今社会，壁画作为一个既有历史意义又有当代艺术价值的艺术形式，其创作及历史保护就显得尤为重要！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 项目介绍\n",
    "\n",
    "&emsp;&emsp;本项目受[DS变沙画-PaddleHub的迁移训练style Transfer图像迁移](https://aistudio.baidu.com/aistudio/projectdetail/2012947?channelType=0&channel=0)启发，基于paddlehub中的预训练风格迁移模型msgnet进行迁移学习。msgnet是基于'Multi-style Generative Network for Real-time Transfer'的风格迁移模型，它能够对输入图片变换21种不同风格。具体可参见[msgnet](https://www.paddlepaddle.org.cn/hubdetail?name=msgnet&en_category=ImageEditing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 代码步骤\n",
    "\n",
    "&emsp;&emsp;本项目基于PaddlePaddle 2.0.2,PaddleHub 2.0.4，其中PaddleHub 2.0.4为AIStudio平台预装版本，故无需额外更改PaddleHub版本。（其他PaddleHub版本可能报错，各位小伙伴要是在其他环境体验要记得检查自己的PaddlePaddle和PaddleHub版本嗷！）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Step1: 安装msgnet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!hub install msgnet==1.0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Step2: 相关数据准备\n",
    "&emsp;&emsp;在完成PaddlePaddle与PaddleHub核对，以及msgnet安装后，即可开始使用msgnet模型对MiniCOCO等数据集进行Fine-tune。下面解压本项目挂在的作为style的壁画图片数据集，并下载minicoco作迁移训练数据。\n",
    "\n",
    "【关于数据集说明】壁画数据集为本人百度爬虫搜集而来，并做了相应清洗，有325张壁画图片。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\r\n",
    "if not os.path.exists('data/minicoco'):\r\n",
    "    %cd /home/aistudio/data/\r\n",
    "    !unzip -oq /home/aistudio/data/data102159/mural.zip -d mural\r\n",
    "    !wget -q https://paddlehub.bj.bcebos.com/dygraph/datasets/minicoco.tar.gz\r\n",
    "    !tar -zxvf  /home/aistudio/data/minicoco.tar.gz\r\n",
    "    %cd /home/aistudio/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "&emsp;&emsp;获取本项目所需的相关数据后，需要对数据进行处理，并按Paddlepaddle2.0 的格式建立dataset。本项目将整个mural文件夹中的图片作Style，但依据[沙画项目](https://aistudio.baidu.com/aistudio/projectdetail/2012947?channelType=0&channel=0)，单张图作style，而非整个所有图片作Style时loss下降较快。若要整个文件夹训练而不是指定某张图片，把styleImg=None即可。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\r\n",
    "from typing import Callable\r\n",
    "import cv2\r\n",
    "import paddle\r\n",
    "import paddlehub as hub\r\n",
    "import numpy as np\r\n",
    "import paddlehub.vision.transforms as T\r\n",
    "import paddlehub.env as hubenv\r\n",
    "from paddlehub.vision.utils import get_img_file\r\n",
    "import os\r\n",
    "\r\n",
    "class MyMiniCOCO(paddle.io.Dataset):\r\n",
    "    \"\"\"\r\n",
    "    Dataset for Style transfer. The dataset contains 2001 images for training set and 200 images for testing set.\r\n",
    "    They are derived form COCO2014. Meanwhile, it contains 21 different style pictures in file \"21styles\".\r\n",
    "    Args:\r\n",
    "       transform(callmethod) : The method of preprocess images.\r\n",
    "       mode(str): The mode for preparing dataset.\r\n",
    "    Returns:\r\n",
    "        DataSet: An iterable object for data iterating\r\n",
    "    \"\"\"\r\n",
    "    def __init__(self, transform: Callable, mode: str = 'train',\r\n",
    "    data1path='/home/aistudio/data',styleFolder='mural',styleImg=None):\r\n",
    "        self.mode = mode\r\n",
    "        self.transform = transform\r\n",
    "        if self.mode == 'train':\r\n",
    "            self.file = 'train'\r\n",
    "        elif self.mode == 'test':\r\n",
    "            self.file = 'test'\r\n",
    "        if styleImg:\r\n",
    "            self.style_file =os.path.join(data1path,  styleFolder,styleImg)\r\n",
    "            self.style=[self.style_file]\r\n",
    "        else:        \r\n",
    "            self.style_file =os.path.join(data1path,  styleFolder)\r\n",
    "            self.style = get_img_file(self.style_file)\r\n",
    "\r\n",
    "        self.file = os.path.join(data1path, 'minicoco', self.file)\r\n",
    "        self.data = get_img_file(self.file)\r\n",
    "        assert (len(self.style)>0 and len(self.data)>0)\r\n",
    "        print('self.data',len(self.data))\r\n",
    "        print('self.style',len(self.style))\r\n",
    "\r\n",
    "\r\n",
    "    def getImg(self,group,idx):\r\n",
    "        im=[]\r\n",
    "        ii=idx\r\n",
    "        while len(im)==0:\r\n",
    "            try:\r\n",
    "                \r\n",
    "                im = self.transform(group[ii])\r\n",
    "            except :\r\n",
    "                \r\n",
    "                print('v',len(group),ii)\r\n",
    "            ii-=1\r\n",
    "        return im\r\n",
    "    def __getitem__(self, idx: int) -> np.ndarray:\r\n",
    "\r\n",
    "        im = self.getImg(self.data,idx)\r\n",
    "        im = im.astype('float32')\r\n",
    "        style_idx = idx % len(self.style)\r\n",
    "        style = self.getImg(self.style,style_idx)\r\n",
    "        style = style.astype('float32')\r\n",
    "        return im, style\r\n",
    "\r\n",
    "    def __len__(self):\r\n",
    "        return len(self.data)\r\n",
    "\r\n",
    "transform = T.Compose([T.Resize((256, 256), interpolation='LINEAR')])\r\n",
    "\r\n",
    "styledata = MyMiniCOCO(transform,mode='train',data1path='/home/aistudio/data',styleFolder='mural',styleImg=None)\r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Step3: fine-tune训练\n",
    "\n",
    "\n",
    "【细节部分】：  \n",
    "\n",
    "1、加载预训练模型：  \n",
    "\n",
    "`model = hub.Module(name='msgnet',load_checkpoint=checkpoint)`  \n",
    "\n",
    "其中：\t  \n",
    "\n",
    "&emsp;&emsp;* name: 选择预训练模型的名字。  \n",
    "\n",
    "&emsp;&emsp;* load_checkpoint: 是否加载自己训练的模型，若为None，则加载提供的模型默认参数。\n",
    "\n",
    "2、 选择优化策略和运行配置   \n",
    "```\n",
    "import paddle\n",
    "from paddlehub.finetune.trainer import Trainer\n",
    "\n",
    "optimizer = paddle.optimizer.Adam(learning_rate=0.0001, parameters=model.parameters())\n",
    "trainer = Trainer(model, optimizer,use_gpu=True,use_vdl=True, checkpoint_dir='test_style_ckpt')\n",
    "trainer.train(styledata, epochs=200, batch_size=32, eval_dataset=None, log_interval=10, save_interval=1)\n",
    "```\n",
    "详情可见[官方PaddleHub finetune风格迁移教程](https://aistudio.baidu.com/aistudio/projectdetail/2231604)  \n",
    "\n",
    "3、 额外说明\n",
    "\n",
    "&emsp;&emsp;由于backbone部分其实不需要迁移，有可能影响迁移效果，故而下载paddle.vision中的vgg16，打印看看他有26组参数，所以设置msgnet的第25个之前的参数stop_gradient=True。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle\r\n",
    "import paddlehub as hub\r\n",
    "from paddle.vision.models import vgg16\r\n",
    "from paddlehub.finetune.trainer import Trainer\r\n",
    "\r\n",
    "import paddlehub.vision.transforms as T\r\n",
    "# paddle.set_device(\"gpu:0\")\r\n",
    "import os\r\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0\"\r\n",
    "# %cd /home/aistudio/\r\n",
    "# backbone model\r\n",
    "backbone = vgg16(pretrained=True,num_classes=-1)\r\n",
    "print('vgg parameters nums:',len(list(backbone.named_parameters())))\r\n",
    "del backbone\r\n",
    "##\r\n",
    "trainFlag=True\r\n",
    "goOnTrain=True\r\n",
    "\r\n",
    "model = hub.Module(name='msgnet',load_checkpoint=None)\r\n",
    "    \r\n",
    "print(type(model),' parameters nums:',len(model.parameters()))\r\n",
    "    ##\r\n",
    "for index,param in enumerate(model.parameters()):\r\n",
    "     #print(param.name)        \r\n",
    "    if index>25:\r\n",
    "            param.stop_gradient=False\r\n",
    "    else:\r\n",
    "        param.stop_gradient=True\r\n",
    "\r\n",
    "optimizer = paddle.optimizer.Adam(learning_rate=0.0001, parameters=model.parameters())\r\n",
    "trainer = Trainer(model, optimizer,use_gpu=True,use_vdl=True, checkpoint_dir='test_style_ckpt')\r\n",
    "trainer.train(styledata, epochs=200, batch_size=32, eval_dataset=None, log_interval=10, save_interval=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Step4: 模型保存\n",
    "\n",
    "通过查看msgnet的module.py源码，我们需要修改之前finetune训练的模型名称。\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/acb54fb94906452c8899341f547732949f0b88a6d4f2414d8438be41524a3693)\n",
    "以下是修改模型名称的脚本，这里讲训练的第200个epoch转换保存了，具体的可以修改："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def epochModel():\r\n",
    "    !mkdir /home/aistudio/model\r\n",
    "    !rm -rf /home/aistudio/model/msgnet\r\n",
    "    !mkdir /home/aistudio/model/msgnet\r\n",
    "    ## 看需要取哪个epoch生成模型\r\n",
    "    !cp -r test_style_ckpt/epoch_200/* /home/aistudio/model/msgnet/\r\n",
    "    ##\r\n",
    "    !mv /home/aistudio/model/msgnet/model.pdopt  /home/aistudio/model/msgnet/style_paddle.pdopt\r\n",
    "    !mv /home/aistudio/model/msgnet/model.pdparams  /home/aistudio/model/msgnet/style_paddle.pdparams\r\n",
    "epochModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Step5: 模型预测\n",
    "\n",
    "模型准备完毕后就可以使用自己训练的模型来体验啦！\n",
    "\n",
    "origin里是原图地址，style是作style的图片地址，save_path是经过风格迁移后的图片保存地址"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle\r\n",
    "import paddlehub as hub\r\n",
    "model = hub.Module(name='msgnet')\r\n",
    "model = hub.Module(directory='model/msgnet')\r\n",
    "result = model.predict(origin=[\"test1.jpg\"], style=\"style_demo_img1.jpg\", visualization=True, save_path ='style_tranfer')\r\n",
    "result = model.predict(origin=[\"test1.jpg\"], style=\"style_demo_img2.jpg\", visualization=True, save_path ='style_tranfer')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 参考：\n",
    "１. [官方PaddleHub finetune风格迁移教程](https://aistudio.baidu.com/aistudio/projectdetail/2231604)　　　　\n",
    "\n",
    "２. [DS变沙画-PaddleHub的迁移训练style Transfer图像迁移](https://aistudio.baidu.com/aistudio/projectdetail/2012947?channelType=0&channel=0)　　\n",
    "\n",
    "３.[Paddlehub msgnet官网](https://www.paddlepaddle.org.cn/hubdetail?name=msgnet&en_category=ImageEditing)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 2.0.0b0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
